ANALYTICS ENGINEERING / SERVERLESS ELT
A Medallion ELT pipeline that ingests Vietnam gold prices daily on GitHub Actions, models Bronze–Silver–Gold layers with dbt on BigQuery, and publishes trends, brand spreads and day-over-day volatility to a self-updating README dashboard — with Discord alerts when things move.
01 / CONTEXT & PROBLEM
Vietnamese gold prices vary by brand, and the questions people actually ask — how has the spread between brands moved, how volatile was this week, is today's jump unusual — need accumulated history and modeled comparisons. The constraint I set: the whole loop must run unattended and cost nothing, using GitHub Actions as both the scheduler and the compute.
02 / ARCHITECTURE
Price sources
8 VN gold brands
→
Python ingestion
daily GitHub Actions schedule
→
Bronze on GCS
raw immutable snapshots
→
dbt on BigQuery: Silver → Gold
cleaned history → trends · spreads · volatility
→
README dashboard + Discord
self-updating charts · movement alerts
03 / KEY DECISIONS & RESULTS
GitHub Actions as the entire runtime
No VM, no orchestrator to patch: the scheduler, compute and deployment are one workflow file. The trade-off — no long-running processes — fits a once-daily batch perfectly.
README as the dashboard surface
The repo page is where visitors already land, so the pipeline regenerates its charts there on every run — the project demos itself with zero hosting.
Immutable Bronze snapshots
Every day's raw payload lands untouched in GCS before any transformation — modeling bugs are re-runs against preserved history, never data loss.
04 / WHAT I'D DO DIFFERENTLY
Public price pages change without notice; the pipeline should detect schema drift and alert on missing brands rather than silently thinning the history. Source freshness tests and a dead-letter path for malformed snapshots are the next additions.